09/830299 

JC02 Rec'c XVPIO 2 5 APR 2001 

18390.4 

Translation of PCT/EP99/08078 as filed on October 26, 1999 

\' 

Interactive Film Reproduction 

There have been various suggestions for providing users with 
interactive film or program replay. It has been proposed e.g. 
to provide a film with a happy or sad ending and to allow the 
viewers in the cinema to decide by majority which content the 
film is to have such that all cinema visitors see either the 
SB happy or sad ending. 
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It is moreover known to record an event, e.g. a car race, 
using many different cameras, e.g. cameras outside of the 
track and cameras which are disposed in the cockpits of the 
respective race drivers. The respective camera images are 
transferred simultaneously to the TV receivers - e.g. the 
digital TV DFl - and the viewer can select, from a plurality 
of possible camera settings, the one which he/she prefers at 
the moment, e.g. the camera in the cockpit of Michael 
Schumacher . 

Finally, it has been proposed that the platform in a 3D 
cinema on which the viewer stands or sits, is movable with 
the motion being controlled by the replayed film content. 



The ''Zentrum fiir Kunst und Medientechnologie Karlsruhe (ZKM) 
has furthermore proposed, in an inflatable hemisphere in 
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which a 3D projection system is installed, displacement of 
the projected image to any position on the inside of the 
hemisphere. The position of the image is thereby linked to 
the head movement of a visitor and follows his/her viewing 
direction. In contrast to e.g. cinema or TV with which image 
reproduction is always limited to a certain individual 
location, namely, a section of the cinema screen or TV 
screen, the above mentioned method of 3D projection displaces 
an image within the hemisphere to a location which the 
observer currently views . 

It is moreover known to use the above-described 3D projection 
system within a hemisphere for virtual travel through a 
reconstructed landscape. Through selecting (clicking on) 
certain image units, the viewer himself can determine his/her 
own "walk"', while he/she does not actually move and always 
remains at the same location. 

It is the purpose of the present invention to provide an 
interactive film perception which is inexpensive, attractive 
and which preferably permits the viewer to simultaneously be 
director and camera man during replay of the film. 

The object in acct>a;dance with the invention is achieved by a 
method in accordance wi<h claim 1. Advantageous further 
developments are describedNm the dependent claims. 



nvention is based on the following considerations: A 
person lives in a three-dimensional environment and, in 



reality, always senses (at least partly) even those things 
which are outside of his field of vision or which he/she can 
sense less sharply. A simple example is the driver of a car 
who drives on a road with his eyes directed forward. If a 
following driver sounds the horn, the preceding driver will 
hear that sound. He/she thereby senses what is behind him/her 
and outside of his/her field of vision and will immediately 
look (into the rear mirror) to see what is going on behind 
him/her to draw the associated conclusions. Just as a look 
into the rear view mirror shows only a small window of what 
is going on behind the driver without the driver being able 
to recognize what is happening outside of the mirrored 
surface, a preferred embodiment of the invention provides 
that only a film section - also called a window - of the 
entire film is visible while all other parts of the film 
outside of the window are not replayed or are darkened and 
are therefore not or only barely visible. The movement of the 
window permits the viewer to see all parts of the film, 
however, not at the same time. The "viewing window", i.e. the 
visible film section has e.g. a length-side-ratio of 4 to 3, 
16 to 9, or any other shape. 

In the cinema and on TV and in all known replay forms of 
moving images, the viewer is always presented with only a 
narrow camera section (irrespective of whether the camera is 
stationary or movable) , wherein the camera section has a 
regular width such that it can largely be detected by the 
field of vision and thus the reception capacity of a person. 
In contrast to the three-dimensional perception of a person 



in his/her surroundings, the usual film and TV replay is 
directed to one single dimension. Even if the user can select 
different camera positions, this only results in 
multiplication of one dimension without providing a three- 
dimensional sensation which is close to reality. 

Finally, the invention is based on the finding that visual 
perception depends largely on orientation and, vice versa, 
that the visual sense also influences sensing of acoustical 
information . 

In a movie, the offered images are usually matched to sounds 
also describing the image, with the exception of background 
sound, e.g. film music, to influence the mood of the film. If 
there is a noise in a film which is not matched to the image 
content shown, the camera will usually move immediately after 
that, sound to the possible source of the sound such that the 
viewer gains a visual explanation for the preceding sound. 

In the above-mentioned film replays, the viewer is largely 
bound to the camera work and does not know what is happening 
outside of the section recorded by the camera. 

The invention describes a new interactive fonna of film replay 
and sensation wherein the viewer and perceiver is offered an 
image which is larger than his/her field of vision. Although 
the field of vision of a person is theoretically 180°, actual 
perception is, however, limited to a field of vision which is 
considerably less than the theoretical, e.g. 70°. Since a 



person can move his/her body and head to direct his/her field 
of vision towards any point of the surroundings, no handicap 
results rather advantages, since the human brain must only 
process that visual information present in the central vision 
field. 

The invention furthermore provides that the viewer is coupled 
to an orientation unit which detects the viewing direction of 
the observer. If the viewer watches an individual scene of a 
film consisting of panoramic scenes, the audio and/or visual 
reproduction of this scene is emphasized compared to the 
other scenes. Such emphasis may e.g. be realized by replaying 
acoustical infoormation from the viewed scene in a louder or 
clearer manner than from the other scenes which are not 
currently being viewed or which are not visibly replayed. One 
individual scene can. also be emphasized by enlarged replay of 
the viewed scene compared to the other scenes or with e.g. 
higher resolution and sharpness of the viewed scene compared 
to the other scenes not currently observed by the viewer. In 
particular, the audio information from the scenes outside of 
the viewing direction of the viewer is not completely 
suppressed but can be perceived more or less clearly by the 
viewer . 

The invention permits a sensing of the film which is close to 
reality. An additional particular advantage of the invention 
is the creation of a completely new type of cinema, wherein 
the user, at first viewing, can only see parts of the entire 
film and two viewers could have completely different opinions 



about the film, since they have seen and heard completely 
different portions of the entire film. 

In a particularly preferred fashion, the invention can be 
perceived in a 3D projection system of a hemisphere or full 
sphere. The image recorded from only one camera location can 
be replayed onto the inner surface of the hemisphere using a 
3D projection system. If a corresponding section from the 
surroundings has been recorded with the camera ~ i.e. not 
only the usual narrow camera section but all views which can 
be detected from the camera location - the viewer is 
simultaneously offered many more scenes, in one single image, 
than he can theoretically (visibly) see, since the field of 
vision of the observer is considerably smaller than the 
overall replay surface. This observer burden is further 
increased by simultaneously replaying several different 
scenes, wherein the viewer is usually accustomed to 
concentrating on only one scene and cannot completely detect 
and digest the other scenes. When the viewer looks at a 
certain scene, the replay device is informed thereof by the 
viewing direction orientation unit and the audio information 
which belongs to that currently viewed scene can be sensed 
and is cognizant to the viewer. At the same time, the audio 
information of other scenes is not completely faded out but 
somewhat suppressed (as in real perception) such that the 
observer still notices that something else is going on 
somewhere else in the film, but is not completely cognizant 
of the film scene not viewed. 



In addition to known sound reproduction devices, such as 
headphones and loudspeakers, sound reproduction devices are 
also suitable for an acoustical perception close to reality 
as are known e.g. from WO97/41709. This sound reproduction 
device is particularly preferred since sound perception 
includes the body sound functions and, in contrast to a 
loudspeaker system, several users can have different sound 
perception in one room at the same time and without mutual 
disturbance should e.g. neighboring viewers look at different 
film scenes. Further advantages of the invention can be 
5 extracted from the following exemplary description of the 
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s'^'l figures : 

shows a view of a scene example; 

shows a planar representation of scene 1; 

shows a scene representation alternative to that of 
Fig. 1; 

shows a cross -section through an inventive cinema,- 

shows a view of a replay of an inventive film; 

shows a view from above of an inventive cinema 
arrangement . 

fig. 1 IS a top view of a constructed panoramic film scenery 
1, wherein the entire scene is composed of several scene 
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sections. The first scene section 2 is a road on which a car 
is travelling. The second scene section 3 is a small group of 
trees (park-like grounds) with a dog walking and barking. The 
third scene section 4 is a small pond with quacking ducks 
looking for food. The fourth scene section 5 is a playground 
with several loudly playing children. The entire scenery is 
illuminated by the sun at location 6. All scene sections have 
exactly the same light conditions and the same sun light 
impinging angle. All film scenes are recorded in total by a 
camera 7 at the camera location 8. The camera device 7 
records the entire film scenery, i.e. a maximum of a 360^, 
round view. In addition to the optical recording, the 
respective sounds which belong to a scene are recorded, with 
the recording being carried out such that an optical scene is 
associated with the associated sound during recording. 

Fig. 2 shows the replay of the scene shown in Fig, l in 
planar elevation between 0 and 360° (in the horizontal and 
vertical direction) . Shown, from the left part of the replay 
to the right edge of the replay screen, are the car scene 1, 
then the tree scene 2, then the pond scene 4 and then the 
playground scene 5. The planar replay shows a section G which 
characterizes the area which is usually visible to the human 
eye (field of vision) . It is to be understood that this 
section G always moves with the eyes of the viewer. If the 
viewer looks at the car scene, he/she will clearly hear the 
sound of the passing car, and may also simultaneously hear 
sounds from other scenes. The sounds from the other scenes 
are replayed with a lower volume and less clarity to allow 
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the viewer to experience the film in a manner which is as 
close as possible to real perception. A person standing at 
location 8 of the camera 7, and looking at the road, would 
also hear the sound of the other scene sections without being 
able to perceive them with full optical clarity. This 
experience is also possible with the inventive replay shown 
in Fig . 2 . 

Fig. 4 shows a cross-section of a view onto a closed circular 
surface of a hemisphere constituting the cinema screen. The 
viewer is in the center of the cinema and directs his/her 
eyes towards the film scene 2 - a road with a passing car. 
The viewer wears headphones or a sound reproduction device 
which can be disposed on the head of the viewer, as disclosed 
e.g. in WO97/41709. Moreover, a signal transmitter, e.g. a 
magnetic field generator or a device for generating an 
electromagnetic wave (e.g. an (infrared) light/laser source) 
is disposed on the headphones, on the sound reproducing 
device, or at another location on the viewer's head. A 
uniquely directed signal 23 is transmitted by the signal 
transmitter, e.g. an oriented magnetic field or an oriented 
light beam, wherein the direction of the signal corresponds 
to the head orientation/viewing direction of the cinema 
visitor (viewer) . This can be achieved by disposing the 
signal emitter (mounting) on the head of the user or on the 
headphones in an unambiguous fashion. 

A signal receiving device 24 is also provided which can 
detect the signal 3 or the direction of the signal. If the 



10 



signal transmitter is a magnetic field generator, the 
receiving device can determine the direction of the magnetic 
field of the magnetic field generator. If the signal 
transmitter is an infrared laser light source, the receiving 
device may be a corresponding camera device by means of which 
the direction of the laser light source can be received 
(infrared is invisible to humans) . The receiving device can 
also be movable, or be stationarily disposed in the cinema 
such that it is not visible to the viewer. 

The direction of the signal (e.g. 10°) from the signal 
transmitter is determined by the receiving device 24 and, 
depending on the result, the replay is influenced in the 
above-described fashion such that e.g. noises which belong to 
the car scene 2 viewed are reproduced at a higher volume than 
noises from other scenes not currently observed by the 
viewer . 

In addition to the direction of the transmitted signal from 
the signal transmitter, the receiving device may also 
determine the exact location of viewing. If the viewing 
location is not in the foreground of the film scene but 
rather in the background, the background will also be given 
sharp contrast and/or be enlarged (PAN/-Zoom) as much as 
possible such that the viewer can perceive not only a two- 
dimensional but a real three-dimensional image. 

If the car scene shows e.g. a mother walking with her child 
on the pavement behind the road (Fig. i) , and the viewer 



directs his/her view onto this lower section of the scene, 
he/she will perceive what the mother says to her child or can 
recognize the person and her clothes as exactly and sharply 
as possible. 

Fig. 3 shows a view of scene other than that of Fig, i. The 
scene shown in Fig. 3 is a restaurant with many tables, a 
restaurant entrance, a counter (bar) , a cabaret stage and a 
bordering kitchen. The camera location is again characterized 
with position 8 and the camera 7 records the entire 
surrounding scene. A couple is seated at table Tl, three 
friends sit around table T2 , a member of the board of a 
company sits at table T3 , four bowling companions sit at 
table T4, a family with a child sits at table T5 , at table T6 
is a group from regulars of a students association, an 
attorney and his clients sit at table T7 , at table T8 is a 
children Vs birthday party, at table T9 are three criminals 
planning a bank robbery and the last table TIO is occupied by 
a group of politicians. The bartender at the counter prepares 
the ordered drinks and individual persons leave or enter the 
room through the entrance, from which street noises can be 
heard. The table arrangement suggests that every individual 
table has different topics for discussion. If, after several 
minutes of this recording, there is an explosion in the 
kitchen, all people in the restaurant will look in the 
direction of the kitchen, and each individual could perceive 
or interpret the explosion differently. The camera 
simultaneously detects all conversations and reactions at all 
tables and one can imagine that this panoramic film recorded 
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by the camera must be replayed several times to see all 
scenes and to digest their content. 

When recording the film scene, the respective optical 
impressions of an individual table are associated with the 
respective acoustical information of that table such that, 
during replay, a corresponding amplification of the audio 
information is possible through replay of other acoustical 
information from other tables. All optical or acoustical 
information can also be initially associated with directional 
information (e.g. in degrees) to audio-visually emphasize one 
scene section compared to other scene sections during replay. 

Since every single table effectively represents an individual 
film, the viewer may have to see the same film at least ten 
times in order to experience the entire film, while 
concentrating on different tables each time. It is thereby 
particularly interesting to watch the different reactions to 
the explosion in the kitchen at the various tables, which 
permits interesting character studies. 

Fig. 5 shows a planar view of a screen with a visible section 
of the film shown as rectangular window. A light spot in the 
center of the window comes from a laser (laser pointer) which 
is mounted to the head of the user such that the direction of 
the laser approximately coincides with the viewing direction. 
The visible film section SA (SA„, SAJ is generated by a 
projector and a camera is mechanically and/or electronically 
coupled to the projector to detect the light spot. If the 
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viewer changes the viewing angle, the position of the light 
spot also changes. The camera follows the light spot and 
causes, through control of the projector, the visible section 
window of the film to be displaced in the direction of the 
light spot until the light spot is preferably once more 
within the visible film section. Displacement of the section 
also changes the film content which always depends on that 
content which is to be replayed at the current sectional 
position . 

In general, there are several ways in which the visible 
section (the visible window) of the film can be moved. The 
film projection device can e.g. track until the light spot is. 
centrally located in the window or within a certain central 
region of the window (to avoid constant movement of the 
window) . It is also possible to displace the window until it 
has approached a certain separation from the light spot. It 
may furthermore be advantageous to prevent window movement if 
the light spot is outside of the camera section KA. 
Alternatively, the window can move in the direction in which 
the light spot has left the camera section KA. 

Advantageously, the camera section KA, i.e. the horizontal 
and vertical camera section KA„ and KA„ is larger than the 
window section such that the camera can follow the light spot 
even when it is already outside of the window section SA. 

Fig. 6 shows a cinema arrangement from above, wherein the 
cinema screen has a cross section of a full or partial circle 



40. A projector 41 is located at a central position in the 
cinema and projects the sectional window 42 of the movie 
which is to be viewed onto the screen 40. The width of the 
visible film section (window width) is characterized by SA„. 
Moreover, there is a camera 43 above or below the projector 
or at another location in the cinema which records a 
relatively large camera section KA^, wherein the visible film 
section lies within the camera section. If the film viewer, 
who carries an optical signal transmitter (see description of 
figure 4), looks at the observed film section 42, a light 
spot 44 will appear in the center of the film section. This 
light spot is detected by the camera 43 and the camera 43 
tries to substantially follow all movements of the light 
spot. The camera 43 controls the projector 41, disposed on ^ 
the movable platform, such that when the light spot is 
displaced, the visible film section is also displaced and. the 
film content simultaneously changes. The viewer is thus both 
camera man and film director. 



